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Foreword 

In  April  2002,  the  Army  Training  and  Leader  Development  Panel  (ATLDP)  released 
the  results  of  its  survey  of  35,000  noncommissioned  officers  (NCOs) .  The  ATLDP 
recommendations  included  the  need  for  regular  assessment  of  Soldiers' technical, 
tactical,  and  leadership  skills.  The  need  for  regular  assessment  of  Soldiers'  skills 
coincides  with  a  series  of  research  programs  on  NCO  assessment  and  development 
executed  by  the  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 
(ARI).  ARI's  research  programs  began  with  an  effort  to  identify  potential  knowledge, 
skills,  and  attributes  (KSAs)  for  future  Soldiers,  Soldier  Characteristics  of  the  21st 
Century  (Soldier  21 ).  They  continued  with  an  effort  to  identify  and  validate  potential 
indicators  of  the  KSAs  for  use  in  junior  NCO  promotion,  Maximizing  2 1st  Century 
Noncommissioned  Officers  Performance  ( NC021 ).  Finally,  the  research  program 
described  in  this  report  provides  a  three-phase  investigation  of  the  issues  and  possible 
resolutions  for  developing  a  viable  Army  assessment  program  that  would  be  linked 
to  NCO  promotions.  This  program  is  Performance  Measures  for  21st  Century  Soldiers 
Assessment  (PerformM21 ). 

Information  on  PerformM21  was  briefed  to  the  Deputy  Chief  of  Staff,  Army  G-l  on 
8  Oct  2003,  the  Director,  Military  Personnel  Management,  Army  G- 1 ,  on  25  Jan  2004,  and 
Chief,  Enlisted  Professional  Development,  Directorate  of  Military  Personnel  Management 
on  13  Nov  2003  and  29  Dec  2004,  as  well  as  the  Deputy  Chief  of  Staff,  Operations  and 
Training,  Training  and  Doctrine  Command,  on  3  Mar  2005.  It  was  briefed  to  the  Sergeant 
Major  of  the  Army  on  28  Jan  2003  and  30  Mar  2004.  It  has  been  briefed  quarterly 
to  senior  NCO  representatives  from  U.S.  Army  Training  and  Doctrine  Command 
(TRADOC),  Office  of  the  Sergeant  Major  of  the  Army,  Office  of  the  Army  G- 1 ,  US. 

Forces  Command  (FORSCOM),  U.S.  Army  Reserve  (USAR),  and  the  Army  National 
Guard  (ARNG)  as  members  of  the  Army  Test  Program  Advisory  Team  (ATPAT). 

The  PerformM21  research  program  is  still  in  progress.  This  special  report  to  the  field 
presents  a  non-technical  overview  of  the  current  status  of  the  program  and  outlines 
the  steps  yet  to  be  taken.  It  is  intended  to  inform,  stimulate,  and  invoke  discussion 
and  response. 


Director  and  Chief  Psychologist 
of  the  United  States  Army 
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Assessing  NCO  Skills 

What  is  Assessment? 

ssessment  is  a  term  that  refers  to  a  standardized,  systematic,  and  objective 
method  for  evaluating  individuals’ performance  in  selected  areas.  Many 
would  refer  to  this  type  of  method  of  evaluating  as  a  “test.”  In  its  application,  it 
centers  on  job  performance  that  can  be  scored  objectively  and  demonstrates  a  level  of 
knowledge  or  skill  an  individual  has  in  a  given  area. 

Why  Do  Assessments? 

The  value  of  assessment  is  in  measuring 
and  validating  knowledge  and  experience 
in  order  to  award  credit.  Assessments 
can  be  used  to  enhance  both  personnel 
and  training  management  practices.  In 
the  personnel  arena,  it  can  help  identify 
who  should  be  promoted  and/or  who 
will  succeed  in  a  special  assignment.  In 
the  training  arena,  it  can  help  validate  skills  and  experiences  and/or  evaluate  training 
needs  and  effectiveness.  Good  assessment  leads  to  good  decisions,  which  leads  to  an 
overall  increase  in  organizational  performance  through  a  better  NCO  corps.  Testing  is 
not  a  replacement  for  development  and  training,  but  it  does  help  ensure  accountability 
for  those  activities.  Assessment  should  drive  training  only  to  the  extent  that  both 
assessment  and  training  are  focused  on  the  same  critical  job  requirements. 

Like  anything  else  however,  not  every  assessment  test  is  equally  good.  To  be  effective, 
assessments  must  reflect  job  requirements  and  frame  questions  in  a  realistic 
manner  that  maximizes  the  relationship  between  the  testing  context  and  on-the-job 
performance.  This  can  be  done  and  there  is  overwhelming  research  evidence  that 
effective  testing  does  improve  individual  and  organizational  performance. 

The  Army  has  a  long  history  of  assessment,  beginning  with  the  introduction  of  group 
testing  during  World  War  I  and  continuing  with  the  current  Armed  Services  Vocational 
Aptitude  Battery  (ASVAB).  But  what  makes  assessing  NCOs  a  current  topic  of  interest? 
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A  Mandate  from  the  Field 

“Develop  and  sustain  a  competency  assessment  program  for  evaluating  Soldiers'  technical 
and  tactical  proficiency  in  the  military  occupational  specialty  (MOS)  and  leadership  skills 
for  their  rank  ” 

This  statement  comes  not  from  the  Army  policymakers  but  from  Soldiers  and  leaders 
across  the  operational  spectrum.  In  2001,  the  Army  Training  and  Leader  Development 
Panel  (ATLDP)  sought  input  from  35,000  NCOs  and  their  supervisors  on  a  variety 
of  topics  to  chart  the  future  needs  and  requirements  of  the  NCO  corps.  This  effort 
is  crucial  because  it  reflects  what  NCOs  see  as  necessary  to  enhance  and  sustain  a 
competent,  professional  NCO  corps  to  meet  current  and  future  demands.  Foremost 
among  those  requirements  was  the  need  for  an  objective  job  assessment  as  additional 
criterion  for  the  selection  of  Soldiers  to  positions  of  leadership  and  responsibility. 

Endorsed  by  the  Chief  of  Staff  of  the  Army,  the  ATLDP  findings  became  the  focus  of 
attention  for  Army  Leadership.  In  2003,  Sergeant  Major  of  the  Army  (SMA)  (R)  Jack 
Tilley  outlined  a  potential  competency  assessment  program  that  should:* 

•  Be  incorporated  into  the  NCO  promotion  system 

•  Be  web -based 

•  Apply  to  the  entire  Army  (Active  Component,  Army  National  Guard,  Army  Reserve) 

•  Apply  to  Specialist/Corporal  (E4)  through  Sergeant  First  Class  (E7) 

•  Be  initially  developed  as  a  core  assessment,  applicable  to  all  Soldiers  regardless  of 
military  occupational  specialty  (MOS);  MOS-specific  competency  assessments  to 
be  added  later 

•  Include  basic  NCO  history,  Army  values,  leadership,  and  training  subject  matter  in 
the  common  core  assessment 

•  Include  situation -based  assessments 


Part  of  a  Continuing  Program 


*  The  assessment  program  and  SMA  Tilley's  guidance  were  briefed  to,  and  endorsed  by,  SMA  Preston  in  March  2004. 
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The  Assessment  Research  Program 

To  meet  the  Army’s  need  for  job-based  measures,  the  U.S.  Army  Research  Institute  for 
the  Behavioral  and  Social  Sciences  (ARI)  started  a  3-year  feasibility  research  program 
to  identify  viable  approaches  for  an  effective  and  affordable  Soldier  assessment  system. 
The  work  is  being  conducted  in  three  phases: 

•  Phase  I:  Perform  a  needs  analysis  and  examination  of  assessment  options. 

•  Phase  II:  Design  and  develop  a  prototype  core  knowledge  assessment. 

•  Phase  III:  Examine  job-specific  measurement  alternatives  and  develop  final 
recommendations  for  an  Army  system. 

Currently  the  3-year  ARI  research  program  is  about  50% 
complete,  and  although  it  is  too  early  to  report  results, 
progress  to  date  has  enabled  us  to  develop  a  vision  of 
what  such  a  program  might  look  like  and  what  the  Army 
would  need  to  include.  As  of  yet,  these  are  initial  concepts 
based  on  our  research  -  it  still  remains  for  Army  leaders  to 
approve  the  elements  of  the  assessment  program  as  well  as 
to  make  a  final  decision  regarding  whether  an  assessment 
program  will  be  implemented.  However,  based  on  our 
research,  the  following  design  features  seem  likely: 

•  The  performance  assessment  system  would  consist  of 
core  assessments  (all  Soldiers  -  Army-wide  skills  and 
knowledge)  plus  technical  (MOS)  assessments,  tested 
separately.  Core  assessments  could  be  implemented 
fairly  quickly  while  technical  assessments  would 
require  a  somewhat  longer  lead  time. 

•  Computer  scorable  items  would  be  designed  to  be  realistic  rather  than  textbook-like. 

•  Tests  would  be  administered  annually  (although  individual  Soldiers  might  only 
have  required  assessments  every  three  years  consistent  with  the  brigade  combat 
team  lifecycle,  or  in  conjunction  with  promotion  eligibility).  The  assessment 
program  would  be  phased  in  gradually  (e.g.,  starting  with  only  one  or  two  pay 
grades).  The  first  assessments  would  most  likely  be  part  of  a  program  pilot  and 
not  be  for-record. 


Soldier  working  on  laptop 
computer;  assessment 
items  are  designed  with 
simulation  to  be  realistic. 


•  Defined  windows  would  be  established  for  test  preparation,  test  taking,  and  test 
make-up  or  retesting.  Emphasis  would  be  on  maximizing  Soldier  access  to  test 
taking  within  the  defined  windows. 
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•  Core  assessments  would  be  scheduled  and  administered  on-line  or  server-based 
through  proctored,  Army-sponsored  portals. 

•  Scores  would  be  converted  to  promotion  points  and  entered  into  Soldier  records. 
Individual  feedback  reports  would  be  provided  to  Soldiers. 

•  Test  preparation  guides  would  be  electronically  available  to  Soldiers  and  would  be 
supplemented  by  a  self-assessment  program  that  refers  Soldiers  to  training  and  job 
aids  as  part  of  their  preparation.  A  self-assessment  program  would  operate  before 
for-record  core  assessments  were  instituted. 


Advantages  of  an  Army  Job  Assessment 


I 
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PERF0RMM21:  Getting  Assessment  Right 


Instituting  an  Army  wide  assessment  program  is  a  significant  undertaking 
involving  virtually  every  aspect  of  Army  functions  -  personnel,  training, 
operations,  communications,  and  computers  -  and  all  of  the  major  Army 
commands.  Moreover,  competency  assessment  itself  is  a  complex  issue  requiring  a 
systematic  foundation  in  job  and  performance  analysis,  development  of  test  plans  or 
blueprints,  tryouts  and  validation  of  items,  and  specifications  for  scoring  and  feedback. 
The  availability  of  computer-based,  computer-administered  testing,  while  affording 
tremendous  new  opportunities,  also  requires  new  paradigms  of  test  development  and 
administration.  Army  assessment  requires  a  collusion  of  many  players  -  test  developers, 
analysts,  subject  matter  experts  (SMEs), policymakers,  trainers,  and  systems  designers 
and  administrators. 


Above  all,  assessment  is  a  serious  and 
sensitive  issue.  Leaders  and  Soldiers  must 
have  confidence  in  their  testing  system. 
Confidence  comes  from  knowledge  and 
understanding  -  failure  to  communicate 
all  aspects  of  the  system  to  the  user  breeds 
misconceptions  and  suspicions.  A  hastily 
developed  assessment  system,  which  has 
not  been  thoroughly  investigated  for  flaws 
and  complications,  cannot  succeed.  A 
competency  assessment  campaign  is  just 
like  any  commander’s  battle  operation  - 
it  must  be  solidly  grounded  in  expert 
input,  detailed  planning,  rehearsals,  and 
anticipation  of  branches  and  sequels. 


Soldiers  working  on  laptop 
computers;  assessment 
involves  many  aspects  of 
Army  functions  -  personnel, 
training  and  development, 
operations,  communications, 
and  computers. 


\  NCOs  in  the  Expeditionary  Army 

\  "As  we  strive  to  develop  flexible,  adaptive,  competent  leaders  it  is  imperative  that  we  have  the  right  tools 
|  to  identify  and  prepare  Soldiers  to  perform,  lead,  and  excel  in  an  Expeditionary  Army.  The  Global  War  on 

|  Terrorism  here  in  Afghanistan  presents  challenges  that  cause  Soldiers  to  use  an  array  of  skills:  problem 

j  solving,  conceptual,  communication,  tactical  and  technical,  interpersonal,  mental,  emotional,  and  team 
\  building.  They  have  to  be  agile,  flexible,  and  adaptive,  and  have  a  keen  sense  of  situational  and  self- 
awareness.  We  must  find  ways  to  identify  and  assess  potential,  and  evaluate  leadership  and  performance  so 
j  that  we  can  teach,  coach  and  mentor  them  to  even  higher  performance.  More  than  ever,  we  need  to  ensure 
I  our  Soldiers  have  the  capabilities  to  lead,  perform,  and  excel  in  an  Expeditionary  Army. " 

f  CSM  Cynthia  Pritchett 
j:  Command  Sergeant  Major 

;  Headquarters,  Combined  Forces  Command  -  Afghanistan 

E: 
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Many  remember  the  Army  experience  with  the  Skill  Qualification  Test/Soldier 
Development  Test  (SQT/SDT)  of  the  1980s  and  early  1990s.  Because  the  lessons  learned 
from  SQT/SDT  programs  are  so  valuable,  a  section  is  devoted  to  this  experience  at  the 
end  of  the  report. 

The  project  takes  a  deliberative  approach  to  researching  the  foundation  of  an  effective 
assessment  system.  An  overview  of  our  approach  is  shown  below.  The  method  involves 
gathering  data  and  information,  and  identifying  and  exploring  options,  but  maintaining 
an  eye  towards  the  operational  implications.  This  allows  design  and  testing  of  an 
operational  system  using  prototypes  to  discover  pieces  that  work,  as  well  as  potential 
pitfalls  for  the  operational  implementation. 


Mm 8 


SoMfet 

Technical 


From  Research  to  Operation 

m  OPERATIONAL  CONCEPT 

Soldier  Competency  Assessment  Program 


-  V%  X  C? : v '* 


SoWiecs  will  get  test  prep  glides 
jfetteNCO 


mw 


As  outlined  by  the  diagram,  there  have  been  three  main  concentrations  of  research. 

1 .  Army- Wide  Core  Assessment 

2.  Soldier  Preparation:  Test  Guide  and  Self-Assessment 

3.  Technical  Assessments 
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The  Army-Wide  Core  Assessment:  A  Prototype  for  All  Soldiers 

In  2004  a  prototype  test  was  designed  to  measure  basic  soldiering  performance  and 
knowledge  requirements.  Approximately  280  computer-based  test  items  (traditional 
multiple-choice,  matching,  ranking,  and  drag-and-drop)  were  developed  for 
administration  to  E4  Soldiers.  The  content  of  the  test  is: 

•  Common  Tasks  Skill  Level  1  (46%) 

•  Common  Tasks  Skill  Level  2  (14%) 

•  Army  and  NCO  History/ Army  Values  (15%) 

•  NCO  Leadership  (13%) 

•  Conduct  of  Training  (12%) 

This  web-based  prototype  test  was  administered  to  571  Soldiers  in  late  2004  using  the 
Army  Digital  Training  Facilities  (DTF)  computer  suites.  Analysis  of  data  from  these 
administrations  is  on-going. 


A  room  of  Soldiers  taking 
the  prototype  competency 
assessment  in  the  Fort 
Riley  digital  training 
facility. 


Although  the  prototype  could  serve  as  the  groundwork  for  an  operational  assessment, 
it  was  developed  without  a  complete  and  systematic  job  analysis  that  would  provide 
the  best  foundation  for  such  an  operational  test.  A  new  test  blueprint  plan,  based  on 
incumbent  and  supervisor  survey  data,  and  prioritized  and  reviewed  by  knowledgeable 
training,  doctrine,  measurement,  and  policy  groups,  needs  to  be  prepared  to  support 
an  operational  test.  Such  a  procedure  would  insure  timeliness  and  relativity  in  a 
changed  operational  climate.  The  next  generation  test  should  also  reflect  technological 
capabilities  including  advanced  media  and  graphics  potentials  to  include  sound  and 
motion  for  more  realistic  and  versatile  test  item  presentations.  These  capabilities  were 
explored  in  the  MOS  technical  phase  described  later  in  this  section. 
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What  Soldiers  Think 

As  part  of  the  prototype  Army-wide  core  measures,  we  administered  a  post-test  questionnaire  and 
conducted  focus  group  workshops  with  Soldiers  who  took  the  test.  There  were  about  375  participants, 
all  in  the  grade  of  E4,  representing  both  Active  and  Reserve  Component  units,  and  from  a  cross  section  of 
combat,  combat  support,  and  combat  service  support  jobs.  Some  results: 

•  Only  about  10%  of  the  Soldiers  had  used  the  Digital  Training  Facility  (DTF)  before  the  test  but 
75%  agreed  that  the  DTF  was  a  good  facility  fortesting. 

'  •  73%  of  the  Soldiers  had  taken  computertests  before.  When  asked  to  choose  between  a  computer 
test  and  a  paper-and-pencil  test,  91%  preferred  computer  tests. 

•  Many  test  items  are  non-traditional;that  is,  something  other  than  4-option  multiple-choice 
questions.  Examples  include  matching,  drag  and  drop,  and  "select  all  that  apply"  responses. 
Overall,  Soldiers  liked  the  non-traditional  items.  Many  responded  that  it  made  the  test  topics 
more  "interestlng'andwere  "different"  from  typical  multiple-choice  test  items. 

•  Soldiers  were  asked  two  questions  about  the  test  content  categories:  (1)  How  effective  is  the 
test  in  evaluating  proficiency  in  this  area?  aid U)  How  well  did  you  do  today  on  this  part  of  the 
test?  The  questipnsWere  on  a  5-point  scalewlth'veryeffective  -  very  well"  and  "effective  -  well” 

■  Fef»n»Wntin9  joints. 
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Although  Soldiers  knew  they  were  taking  part  In  a  research  project  and  the  test  results  would  not  affect 
them  or  be  tepOited,{prb<tbr$:pWt£'ived  that  almostall  appeared  to  take  the  test  seriously.  Based  on 
the  Soldiers'  reactions,  the  demonstration  core  test  was  not  perceived  to  be  an  "easy"test.  However, 
despite  concerns  about  how  well  they  performed,  Soldiers  did  not  blame  the  test.  Their  appraisal  of  the 
effectiveness  of  the  test  content  was,  overall,  wry  positive. 
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The  Criticality  of  Job  Analysis 

;  It  is  an  axiom  in  the  performance  measurement  field  that  you  don't  have  to  do  job  analysis  for  all  the 
j  assessments  being  developed  -  only  for  the  ones  you  want  done  right.  This  applies  to  both  the  core 

;  assessments  (all  Soldiers)  and  to  the  MOS  technical  assessments.  In  its  simplest  form,  job  analysis  tells 
i  you  what  (content)  and  how  much  (weight)  to  include  in  the  assessment. 

I  Effective  job  analysis  for  purposes  of  assessment  measurement  should  be  based  on  up-to-date  and 

i.  representative  survey  data  from  incumbents  and  supervisors,  reflecting  all  venues  in  which  the  job  is 

l  performed.  These  survey  results  then  must  be  analyzed  and  interpreted  by  knowledgeable  job  experts. 

I  Final  content  selection  fortesting  is  a  combination  of  all  known  factors,  applied  by  knowledgeable  and 
;  experienced  reviewers,  and  policymakers  at  a  high  level  of  authority. 

I! 

j  Current  Army  job  analysis  is  in  a  varied  and  inconsistent  state.  Since  the  job  analysis  that  is  being 
j  conducted  is  not  done  with  an  assessment  goal,  revitalization  of  the  Army  job  analysis  program  will 
I  likely  be  a  necessary  start  point  for  the  introduction  of  an  assessment  program.  Assessments  that  are  not 
|  based  on  an  up-to-date  job  analysis  risk  being  irrelevant  to  the  incumbents  and  will  be  seen  as  outside 

j.  of  the  Soldier's  job. 

f 

!' 


Soldier  Preparation:  Critical  to  the  Mix 

Essential  to  a  Soldier  assessment  program  is  the  concept  of  Soldier  preparation. 

Soldiers  must  be  given  information  about  what  will  be  expected  of  them  and  the 
tools  to  help  them  prepare.  For  the  prototype  core  assessment,  we  developed  a  Test 
Preparation  Guide.  This  Guide  -  available  both  electronically  and  in  hard  copy  -  was 
used  as  combination  read-ahead,  study  guide,  and  test  familiarization  tool.  It  includes 
information  about  what  is  covered  on  the  assessment,  sample  items,  and  links  to 
relevant  study  resources  (e.g.,  Soldier  Manuals) .  While  pre-pilot  test  notification 
problems  precluded  a  full  trial  of  the  intended  use  of  the  Guide,  Soldier  reactions  to  the 
usefulness  of  the  Guide  were  highly  favorable. 

We  also  have  completed  a  prototype  self-assessment  system  designed  to  accompany  the 
introduction  of  an  Army  assessment  program.  Such  a  system  would  provide  individuals 
with  a  means  to  judge  their  strengths  and  weaknesses  in  preparation  for  promotion 
and  provide  opportunities  for  learning  how  to  study.  It  would  be  web-based,  accessed 
through  AKO,  with  links  to  other  related  Army  sites  and  sources.  The  prototype  self- 
assessment  system  has  three  components: 

•  An  Information  Component  that  offers  specifics  about: 

-  Testing  program  goals  and  eligibility  requirements 

-  The  test  environment  and  test  process 

-  Test  preparation  including  test -taking  strategies  (this  incorporates  features  of 
the  Test  Preparation  Guide  described  above) 

-  Links  to  a  description  of  the  semi-centralized  promotion  system  including  an 
interactive  promotion  points  worksheet  (PPW) 
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•  A  Knowledge  Component  consisting  of  practice  items  similar  to  those  from  the 
operational  test.  This  would  not  be  a  “practice  test”  because  it  would  not  give  a 
predictive  score,  but  it  would  allow  Soldiers  to  identify  test  topic  areas  that  are 
weaknesses  for  them. 

•  A  Feedback  Component  consisting  of  identification  of  sources  or  study  references. 
Feedback  needs  to  be  specific  to  the  type  of  item.  Straightforward  procedural 

or  knowledge  items  need  minimal  explanations,  primarily  just  references  to 
the  source  documents.  Leadership,  situational,  and  problem  solving  exercises, 
designed  to  tap  Soldier’s  critical  thinking  skills,  however,  need  more  detailed 
feedback  including  reasons  and  rationales  supporting  choices,  along  with 
principles  and  doctrine  that  govern  performance.  Overall,  the  goal  is  drive 
Soldiers  to  use  training  materials  and  resources  that  already  exist.  These  materials 
and  resources  will  be  provided  as  on-line  links. 

The  prototype  self-assessment  is  being  designed  to  meet  the  immediate  needs  of  the 
initial  Army-wide  core  assessment.  As  the  assessment  system  matures,  it  will  be  capable 
of  adjustment  to  include  providing  MOS-specific  information,  sample  job  measures, 
and  promotion  status  or  comparisons  with  peers  within  the  MOS. 


Thoughts  on  Assessment 


welcome  them  as  a  wayefwtifying  their  knowledge  and  value.' 
SGM  Michael  l  lamb 

TRADOC  Deputy  Chief  of  Staff  for  Operations  and  Training 
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Technical  Assessments:  A  Real-Life  Approach 

An  integral  part  of  the  Army  assessment  program  includes  obtaining  measures  of  a 
Soldier’s  job  proficiency  -  in  short,  technical  testing.  Yet  this  is  the  most  problematic 
aspect.  MOS  vary  greatly  by  incumbent  size,  complexity,  performance  requirements, 
training,  employment,  and  environment.  Even  within  MOS,  there  can  be  differences  in 
utilization,  assignments,  equipment,  and  job  tasks.  Finally,  plans  for  MOS  consolidation 
and  reorganization,  the  increased  use  of  assignment  oriented  training  (AOT),  increased 
reliance  on  additional  skill  identifiers  (ASI),  and  the  very  nature  of  Future  Force  Army 
structure,  indicate  the  turbulence  will  continue  for  the  foreseeable  future. 


Nevertheless,  technical  assessment  is  too 
important  to  be  dismissed  as  “too  hard.” 
One  approach  is  to  throw  out  all  the  old 
ideas  about  technical  assessments  -  mainly 
that  all  MOS,  and  indeed,  all  Skill  Levels 
within  an  MOS,  need  to  be  treated  the 
same.  Since  promotion  competition  is 
by  MOS,  not  between  MOS,  no  issue  of 
fairness  should  exist  if  one  MOS  has  a  test 
and  another  does  not,  or  if  different  MOS 
have  different  assessment  methods,  or  if 
different  MOS  have  different  assessment 
designs  and  cycles.  Precedence  already 
exists  for  requiring  or  rewarding  Army 
jobs  differently  as  part  of  the  promotion 
criteria.  For  example,  the  Ordnance  Corps 
awards  promotion  points  for  civilian 
certifications. 


Soldier  entering  data: 
assessment  will  provide  a 
comprehensive  evaluation. 


As  part  of  the  ongoing  research,  we  investigated  prototype  options  available  to 
assessments  in  five  different  MOS: 


•  63B  Wheeled  Vehicle  Mechanic 

•  91W/68W  Health  Care  Specialist 

•  19K  Abrams  Armor  Crewman 

•  31 B  Military  Police 

•  14E  Patriot  Fire  Control  Enhanced  Operator/Maintainer 
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These  MOS  were  selected  because  they  posed  representative  challenges  and 
opportunities  for  different  assessment  methods.  The  purpose  was  to  explore  the 
assessment  possibilities,  rather  than  to  develop  complete  tests.  Part  of  this  work 
involved  technological  explorations,  including  computer-based  simulations  that 
approximate  job  conditions  without  the  expense  and  demands  of  the  typical  hands- 
on  evaluation  with  real  equipment.  Other  potential  measures  considered  include 
situational  assessments,  job  product  evaluations,  virtual  and  constructive  simulations, 
and  existing  job  measures.  Discoveries  during  this  research  identified  a  number  of 
approaches  that  individual  MOS  could  develop  as  a  means  of  assessment. 


Not  A  Pass  or  Fail  Evaluation 

A  promotion  related  as^n^«jT|afes  Sorters  with  other  similar  SoWiers  in  the  promotion  pool,  not 
with  a  preset  standard  There  Is  n#  score  orWscoring  Involved.  SokHeft  who  perform  much 

better  than  their  peers  wpufcj  to  the  maximum  promotion  points;  allowed  for  the  assessment 

Actor  while  those  to  their  peers wouM&ceive  fewer  promotion  points. 


Whatever  the  outcome,  the  movement  towards  technical  assessments  will  likely  be 
much  slower  than  that  towards  an  Army  core  assessment.  It  is  likely  that  efforts  will  be 
initiated  individually,  by  MOS  or  by  Career  Fields,  rather  than  on  an  Army-wide  scale 
involving  all  MOS.  Indeed,  some  Career  Fields  have  already  made  an  initial  move  and 
many  more  will  follow.  This  independence  and  distinctiveness  in  the  approach  towards 
technical  testing  is  viewed  as  a  positive.  This  does  not  mean  that  development  will  be 
without  boundaries  or  that  practices  can  be  instituted  without  constraint  or  without 
coordination.  But  if  individual  programs  comply  with  professional  testing  standards 
and  with  Army  policy,  they  can  and  should  differ  across  jobs. 


U.S.  Army  Research  Institute 


\  The  Whole  Soldier  Concept 

:  Competency  assessment  results  would  be  another  tool  on  which  to  assess  Soldiers'  readiness  for 
!  promotion  -  but  it  would  not  be  the  sole  tool.  For  the  semi-centralized  promotion  system  (promotion  to 
;  grades  E5  and  E6),  military  and  civilian  education,  awards  and  decorations,  and  military  training  (Army 
i  Physical  Fitness  Test  and  weapons  qualification)  will  continue  as  weighted  factors.  Commanders  would 
!  continue  to  recommend  (or  not  recommend)  Soldiers  for  promotion  and  Promotion  Board  appearances 
may  continue  as  a  requirement.  The  weight  (i.e.,  points)  of  a  competency  assessment  test  within  the 
|  overall  promotion  points  worksheet  is  a  policy  decision  that  is  yet  to  be  determined. 

I  In  the  centralized  promotion  system  (to  grades  E7  and  E8),  competency  assessment  results  become  one 
i  more  factor  available  for  centralized  selection  boards  to  consider.  As  they  do  now,  boards  would  continue 
-  to  receive  specific  guidance  for  promotion  considerations. 


The  Army  Test  Program  Advisory  Team  (ATPAT):  The  Army's  Voice 

As  indicated,  an  Army  assessment  program  encompasses  many  functions  and  facets. 
Research  and  expertise  from  the  testing  community  can  provide  some  of  the  answers 
and  identify  the  design  requirements  for  much  of  a  potential  Army  program.  But  an 
operational  program  will  involve  many  policy  decisions  and  require  organizational  and 
cultural  considerations  that  transcend  purely  test  program-oriented  research.  To  insure 
that  the  research  and  program  development  can  lead  to  an  acceptable,  implementable 
operational  program,  a  special  team,  the  ATPAT,  has  been  in  action  since  April  2003. 
Composed  primarily  of  Sergeants  Major  and  Master  Sergeants  representing  a  wide 
variety  of  Army  commands  and  functions,  the  ATPAT  meets  quarterly  with  the  research 
team.  Although  the  specific  contributions  of  the  ATPAT  are  many  and  varied,  the 
general  functions  of  the  team  are  to: 

•  Serve  as  an  oversight  group  providing  insights  into  operational  implications  and 
real-world  feasibility  of  an  implementable  assessment  program. 

•  Provide  continuing  input  and  update  to  the  analysis  and  definition  of  needs 
requirements  for  an  Army  assessment  program. 

•  Provide  product  reviews,  subject  matter  expertise,  and  external  assistance  and 
contacts  (as  needed)  in  the  process  of  developing  prototype  measures  and  trial 
procedures. 

•  Serve  as  a  conduit  to  explain  the  assessment  research  program  to  various  Army 
agencies  and  constituencies. 

•  Provide  real  time  feedback  on  changing  conditions  and  anticipated  activities 
within  the  Army  that  affect  the  assessment  program. 
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ATPAT  Members 

The  ATPAT  was  established  in  April  2003  largej|t|iqytii tjie  initiative  end  recommendations  of 
CSM  Cynthia  Pritchett,  then  Command  SergearrtMijer,  LJ.S.  Army  Combined  Arms  Center  and  Fort 
Leavenworth.  CSM  Pritchett  served  as  the  ATPAT  Chairperson  until  her  reassign  merit  as  Command 


in  the  ATPAT  as  Chair  Emeritus. 


v. 


There  have 

been  46  ATPAT  members  to  date.  Even  though  some  members  become  inactive,  we  consider  ATPAT 

_ L _ i_«  i _ _ —  — *•  -  . . .  A _ _ : _ ja?— .-  J-  A  J'il  ’  ‘  » 


The  current  ATPAT  thairamon  is  S6M  Mkhitl  1 1  A<ti * " *  ^ ' _ *  »  ...  j 

rv .  n. • ** 


Deputy  Chief  of  Staff  for  Operations  and  Training  (TAfopC,  DCSOPS  4  T)  fort  Monroe,  VA  (757-788-5709) 
michael.lamb#us.army.mJI. 

The  ATPAT  generally  meets  every  three  mpnths-in  Arlington, Virginia.  Participation  in  meetings  is 
sponsored  by  the  memberV  command  oragency. 

More  about  th«  ATPAT 


■=  ' 


team  more  about 


Two  Soldiers  field  training. 
Well-trained  Soldiers 
make  good  decisions, 
which  leads  to  an  overall 
increase  in  organizational 
performance  through 
a  better  NCO  corps. 
Knowledge  gained 
through  training  and 
quality  of  decision  making 
arejust  twoaspects  of 
competency  assessment. 
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Past  Army  Test  Experiences: 

What  We've  Learned 

Testing  for  purposes  of  proficiency  measurement  is  not  a  new  concept  in  the  Army. 
From  the  post-Korean  War  period  to  about  the  mid-1970s,  NCOs  (in  grades  E5 
through  E7)  took  an  annual  MOS  test,  both  to  qualify  in  their  MOS  and  to  draw 
extra  pay  for  the  highest  performers.  This  was  followed  by  the  Skill  Qualification  Test/Soldier 
Development  Test  (SQT/SDT)  period,  which  lasted  into  the  early  1990s.  The  SQT/SDT 
was  very  much  an  evolutionary  process  -  it  was  many  things  over  a  period  of  time,  encom¬ 
passing  written  tests,  hands-on  tests,  certifications,  and  a  variety  of  personnel  applications 
for  scoring.  Almost  everyone’s  recollection  of  SQT/SDT  differs  by  when  they  experienced  it. 

We  have  spent  considerable  time  researching  the  history  of  promotion  testing,  particularly 
the  SQT/SDT  experiences  -  both  the  good  and  the  bad.  We  have  identified  some  lessons 
learned  to  take  forward  into  a  new  Army  testing  program.  Specifically: 

•  Minimize  the  logistical  load.  SQT/SDT  was  a  completely  paper-based,  mail-out,  and 
returns  system. 

•  Do  the  research.  SQT/SDT  broke  a  lot  of  new  ground,  and  very  quickly.  We  know  a  lot 
more  about  large-scale  job  testing  -  in  all  applications  -  than  we  did  30  years  ago. 

•  Perceptions  count  -  whether  they  are  accurate  or  not.  Make  sure  Soldiers,  leaders, 
and  commanders  understand  the  program  and  have  a  way  to  get  answers  to  their 
questions.  Actively  push  information  to  the  field  and  install  programs  that  maximize 
individual  Soldier  contacts. 

•  Minimize  the  administrative  burden,  particularly  those  functions  that  require  Soldiers 
and  in-house  staffing.  SQT/SDT  was  a  task-based,  high  maintenance  system  relying 
almost  completely  on  in-house  test  development  expertise,  resources  and  production, 
plus  SME  augmentation.  Technology,  contracting,  and  new  test  design  strategies  can 
make  a  difference. 

•  Recognize  testing  as  one  part  of  the  whole  Soldier  performance  system.  Testing 
can  enhance  accountability  and  provide  both  individual  and  unit  measures  of 
performance,  but  individual  assessment  testing  needs  to  be  a  part  of  a  larger  Soldier 
development  program  -  testing  is  not  an  end-all  in  itself. 

Some  may  perceive  that  the  SQT/SDT  program  was  a  failure  and,  from  an  operational 
standpoint,  the  logistical  requirements  made  the  program  unwieldy.  From  a  testing 
viewpoint,  however,  it  was  a  highly  successful  program.  Hundreds  of  thousands  of  Soldiers 
were  accurately  and  consistently  measured  in  critical  job  performance  categories.  Low 
performers  were  identified  and  their  skills  were  improved  or  the  Soldiers  were  removed 
from  the  system.  Equally  as  important,  high  performers  were  recognized  and  promoted. 
Soldiers  were  provided  with  specific  expectations  and  leaders  had  a  means  of  measuring 
achievement  of  those  expectations.  In  the  end,  operational  problems,  including  costs  and 
support  requirements,  forced  almost  constant  revision  in  the  program,  and,  ultimately, 
many  of  these  unresolved  factors  motivated  decision  makers  to  abandon  skills  testing  in 
the  Army.  The  intrinsic  quality  and  value  of  the  program,  however,  is  still  recognized,  both 
inside  and  outside  the  Army. 
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Assessment  In  the  Other  Services: 

We  have  spent  some  time  with  the  other  Services  -  particularly  the  Air  Force  and  the  Navy  -  examining  their  promotion  testing 
systems  and  procedures.  We  continue  this  working  relationship  as  a  means  of  exchanging  ideas  that  may  have  Army  applications. 

US  Air  Force  -  The  Air  Force  administers  promotion  examinations  annually  to  all  Airmen  starting  at  E4  and  continuing  for  promotion 
to  E9.  The  system  (called  the  Weighted  Airman  Promotion  System  -  WAPS)  tests  Airmen  with  two  tests  -  a  Promotion  Fitness 
Examination  of  common  tasks,  history,  leadership, conduct, and  customs  and  courtesies  and  a  Specialty  Knowledge  Test  that  Airmen 
take  in  their  job  specialty.  Both  tests  are  knowledge  tests  and  are  currently  paper-and-pencil-based.  Airmen  are  provided  Study 
Guides  and  references  but  all  study  and  preparation  is  an  individual  Airman's  responsibility;  group  preparation  is  prohibited.  Test 
development  is  centralized  at  Randolph  Air  Force  Base, TX  at  a  Flight  with  about  20  miiitary  and  35  civilian  members  permanently 
assigned.  These  are  augmented  on  ah  annual  basis  byen  listed  subject  experts  who  are  brought  in  30-day  temporary  duty  periods. 

The  Air  Force  administers  about  200,000 tests  annual^  to  about  104,000  individual  Airmen. 

US  Navy  -  The  Navy  administers  promotion  tests  to  all  Sailors  In  grades  £4  through  E7.  Tests  are  administered  twice  a  year  to  E4  -  E6 
Sailors  and  annually  to  E7  Sailors.  Two  tests  are  administered  -  a  Professional  Military  Knowledge  test  consisting  of  common  subjects 
such  as  personal  safety  and  damage  control,  nuclear, ttologiol,and  chemical,  security, ieadership,tralning, a nd  Navy  programs  and 
policies,  and  an  Occupational  test  which  is  job-specific.  The  |lavy  tens  approximatety  ISO  job  specialties  (ratings)  with  a  different  test 
in  each  pay  grade  for  each  rating.  Test  development  is  centteHltd  at  the  liavy  Advancement  Center  in  Pensacola,  FL  with  about  50 
personnel  assigned  permanent  party.  This  is  augmentedby  ifiout85  ehfttted  subject  experts  who,  until  very  recently,  were  brought 
into  Pensacola  on  2-year  test  writing  assignments.  The  fjargr  attninjsters  about  290,000 tests  each  year.  They test  all  E4  -E6  Sailors 
in  a  3-hour  window  on  the  same  day  in  March  and  in  September  and  aH  E7  Sailors  blithe  same  day  in  January.  The  Navy  has  recently 
started  prototyping  computer  testing.  f 

US  Marine  Corps- Although  it  isnot  directly  used  in  the  Marine  promotionsystem,  the  Marine  Corps  administers  an  annual  test 

CW02,  and  Officers  OTthrough  03.  TbeLairdnationte^^mil  teeap^jiWta,  land  navigation,  nuclear, 

biological,  and  chemical,  physical  fitness,  history,  leadership,  trobp  Information, and  dose  order  drill.  A  hands-on  evaluation  is 
provided;  however,  the  for-record  test  is  a  knowfedge-based  apminatioiL  A  minimum  passing  wbte  of  b  required.  Marines 
who  fail  are  mediated;  however,  the  firfl  time  tesult  Is  the  score of  record.  Scores  are  entered  on  proficiency  markings  (E4)  and 
Fitness  Reports  for  £5  Marines  and  above.  Unit  t«t  perfom|na  is  aggreg|ted  and  commanders  call  obtain  historical  test  records  on 
individuals.  The  MCCS  comes  in  paper-and-pepdlformat  An  dWcttpnk  version,  Witt  Immediate  firing  and  feedback,  is  also  available 
over  distance  learning  (MarineNet)  at  the  Marine  Learning  Resources  Centers.  Proponent  for  the  MCCS  is  the  Marine  Corps  Institute  at 
the  Marine  Corps  Barracks  in  Washington,  DC. 

US  Coast  Guard  -  The  Coast  Guard  is  currently  phasing  into  operation  a  revised  education  and  qualification  program,  Enlisted 
Professional  Military  Education  (E-PME).  Under  this  program  all  members  of  20  rates  (the  Coast  Guard  equivalent  of  MOS)  in  grades 
E4,  E6,  and  E8  will  be  administered  the  Advancement  Qualification  Exam  (AQE)  which  covers  general  military  knowledge  including 
seamanship,  first  aid,  survival,  customs  and  courtesies, leadership  and  supervision,  small  arms,  and  security.  All  membes  must 
complete  a  series  of  rating-specific  hands-on  performance  qualification  tasks  and  members  in  grades  E4  and  E5  also  must  pass  a 
rating  specific  correspondence  couise.  Members  who  meet  all  other  eligibility  requirements  and  satisfactorily  complete  the  E-PME 
and  qualification  requirements  may  participate  in  the  Service  Wide  Exam  for  advancement.  The  Coast  Guard  publishes  official  study 
guides  as  part  of  the  E-PME  as  well  as  a  checklist  of  all  performance  and  knowledge  requirements  for  each  pay  grade.  Members 
are  encouraged  to  study  collectively  and  there  is  local  study  in  the  unit.  Test  development  is  centralized  at  three  Training  Centers  in 
Petaluma,  CA,  Cape  May,  NJ,  and  Yorktown,  VA.  Subject  matter  experts  assigned  to  the  Training  Centers  on  3-4  year  tours  develop  the 
tests  as  part  of  their  duty  assignment.  The  Advancement  Branch  of  the  Personnel  Service  Center  administers  the  tests. 
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What  Happens  Next? 

The  Short  Term  Outlook  -  The  Long  Term  Vision 


The  PerformM21  project  is  in  progress 
and  much  work  remains  to  be  done.  The 
first  18  months  of  the  project  have  been 
very  productive  in  demonstrating  both 
what  is  needed  and  what  can  be  done 
in  the  area  of  an  Army  core  promotion 
assessment.  Consistent  with  the  “Research 
to  Operation”  concept  shown  on  page  5, 
the  time  is  fast  approaching  for  this  aspect 
of  the  assessment  to  be  handed  over  to  Army  leadership  for  implementation  decisions. 
Meanwhile,  the  second  half  of  the  project  should  see  concentrated  efforts  in  the 
following  areas: 

•  Continuation  of  ATPAT  meetings  on  a  quarterly  basis  to  provide  up  to  date  field 
and  user  input. 

•  Development  of  a  cost  analysis  of  various  implementation  options  along  with  the 
identification  of  potential  benefits  criteria. 

•  Review  and  incorporation  of  operational  requirements  and  lessons  learned  from 
Iraq,  Afghanistan,  and  the  Global  War  on  Terrorism  as  a  basis  of  recreating  the  test 
blueprint  for  the  Army  core  assessment. 

•  Refinement  of  the  specifications  for  technology  and  system  requirements  to 
support  Army-wide  computer-based  testing. 

•  Exploration  of  alternatives  in  MOS  assessment  to  include: 

-  Candidate  jobs  for  full  prototype  development. 

-  Testing  methods  and  enhancements  through  technology  and  low-cost 
simulation. 

-  Situational  judgment  and  decision-making  testing. 

-  Approaches  for  skill  level  implementation  and  job  coverage. 

-  Job  analysis  and  test  content  definition  strategies. 


NCO  using  on-board 
computer;  assessment 
helps  ensure 
accountability  and 
meeting  performance 
standards. 


Planning  for  the  Future:  Progress  Toward  an  NCO  Competency  Assessment  Program 


One  Army 

From  the  outset,  an  operational  principle  of  the  Army’s  new  assessment  program  has  been  that  it 
include,  at  the  start,  all  eligible  Soldiers,  including  AC  as  well  as  RC  (US  Army  Reserve  and  Army  National 
Guard).  All  Soldiers  will  take  the  same  test  and  encounter  the  same  requirements. 


This  does  not  mean  that  there  would  be  a  single  program  or  that  there  would  be  just  one  path  to 
implementation.  The  RC  is  different  in  many  ways  and  preliminary  planning  is  addressing  the  operational 
implications  of  inclusion,  while  still  recognizing  and  working  with  the  unique  factors  affecting  the  RC. 


"We  must  insure  that  the  USAR  and  theARNG  are  included  in  the  planning  and  given  adequate  time  to 
react  to  the  implementation  of  a  testing  proyam,  to  integrate  the  requirements  into  our  training  schedules, 


Components.  It  is  something  we  cm  live  with,  but  it  needs  recognition  and  planning  ’ 


have  caused  concern  about  readiness  in  some  oftiieSold'ier  and  Leadership  skill  sets.  This  test  program 
could  boost  both  the  emphasis  and  accountability  of  performance  requirements  that  are  critical  to  the 
USAR  mission." 


CSM  Nick  Piacentini 

United  States  Army  Reserve  Command  (USARQ 
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The  closing  point  of  our  long-term  vision  is  best  made  by  referring  to  the  work 
performed  by  a  small  group  of  senior  NCOs  who  have  done  some  serious  thinking 
about  the  future  of  the  NCO  corps.  Called  “A  Concept  for  Growing  the  21st  Century 
NCO  Corps,”  this  model  of  development  is  shown  below.  Like  all  concepts,  this  is 
emerging,  subject  to  frequent  updating,  and  does  not  emulate  recent  doctrine  or  policy. 
But  it  does  stimulate  thinking  and  demonstrates  the  need  for  long-term  goals.  It  is 
critical  for  readers  to  comprehend  the  theme  of  changing  requirements  and  career 
development  paths  for  NCOs  depicted  in  the  model.  All  the  elements  portrayed  in 
the  NCO  Development  Model  are  crucial  for  NCO  growth  and  Army  success;  no  one 
element  is  intrinsically  more  important  than  another.  Promotion  testing  is  one  of  those 
elements  and  the  focus  of  our  work.  But  it  is  important  not  to  view  assessment  testing 
as  disconnected,  divergent,  or  an  option.  Like  all  the  other  pieces,  it  helps  make  up  the 
whole  that  is  the  NCO,  and  the  Army  of  the  future  -  a  future  that  has  already  begun. 


NCO  Development  Model 


NCOES  =  Noncommissioned  Officer  Education  System;  CMf=  Career  Management  Field;  CTC = Combat  Training  Center;  ISC  -  First  Sergeant;  BSNCOC = Battle  Staff  Noncommissioned  Officer 
Course;  DS = Drill  Sergeant;  EO = Equal  Opportunity;  MACOM  =  Major  Command;  1C = Inspector  General;  ekrmyii = electronic  Army  University;  SOC = Servicemembers  Opportunity  Colleges; 
SOCAD  =  Servicemembers  Opportunity  Colleges  Army  Degrees;  COOL  =  Credentialing  Opportunities  On-Line;  ACES = Army  Continuing  Education  System;  AKO  =  Army  Knowledge  Online; 
ATIA-M = Army  Training  Information  Architecture -Migrated;  NC0EB= Noncommissioned  Officer  Evaluation  Report;  NCOPD  =  Noncommissioned  Officer  Professional  Development 
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Your  Turn:  Feedback? 


U.S.  Army  Research  Institute 


This  research  program  is  important  to  decisions  the  Army  may  make  that  affect  Soldiers 
in  the  future.  Equally  important  is  that  you  get  a  chance  to  be  heard.  Comments  and 
reactions  are  welcomed.  Please  take  the  time  to  let  us  know  your  thoughts  by  sending 
them  to  ARI,  as  follows: 


Mail: 

Dr.  Tonia  Heffner 
DAPE-ARI-RS 

2511  Jefferson  Davis  Highway 
Arlington,  VA  22202-3926 

Email: 

tonia.heffner@hqda.army.mil 


USAR  Soldier  completing 
the  competency 
assessment  in  a  digital 
training  facility. 


1  Costs  of  an  Army  Assessment  Program 

i  No  change  comes  without  a  price  tag,  and  costs  are  not  always  dollars  and  cents  issues.  Some  associated 
|  testing  requirements: 

I  •  An  Army  infrastructure  to  support  testing  is  required, 
j  •  Soldiers  will  need  time  and  resources  to  prepare  for  testing  and  to  test. 

i 

|  •  Personnel  and  promotion  policies  will  have  to  be  changed;  Soldiers  will  need  to  adjust. 

| 

|  •  Accurate,  up-to-date  study  materials  must  be  made  available. 


\  An  Army  at  War 

:■  When  the  initial  interest  in  Army  job  testing  was  generated  in  2002,  the  operational  commitments  and 
j  priorities  of  the  Army  were  far  different  than  what  exists  in  2005.  Is  testing  a  viable  concept  for  an  Army 
j  at  war?  The  answer  is  YES.  Testing  drives  rapid  dissemination  of  Lessons  Learned  and  tasks  derived 
i  from  combat  experiences.  And,  if  anything,  the  demands  of  increased  operational  tempo  highlight  the 
j  requirement  to  promote  first  those  who  are  best  qualified  to  advance  to  the  NCO  ranks. 

\  Without  doubt,  increased  operational  commitments  and  deployments  involve  special  testing  issues  that 
1  must  be  addressed.  Wartime  demands  may  alter  the  schedule  for  advancing  and  adopting  the  Soldier 
}  assessment  program,  but  competency  assessment  is  a  long-term  program  focused  on  meeting  challenges 
r  and  providing  for  the  continuing  betterment  of  the  NCO  corps.  Development  needs  to  advance  even 
j  when  priorities  are  concentrated  elsewhere  -  the  program  needs  to  be  ready  when  the  Army  is  ready. 

1 
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